feat(retention): chain-aware retention and pruning#8
Conversation
- Add a pure retention engine in internal/dumps (GroupChains + Plan, no I/O or clock): the catalog is grouped into restorable chains (a base plus its incrementals) and kept or pruned as a unit, so an incremental is never orphaned from its base. A chain's age is its newest member, so an actively-appended chain is never pruned mid-life. - Support three composable rules with union semantics (a chain is kept if it satisfies any active rule, so a rule can only ever protect): keep_last N, max_age duration, and GFS daily/weekly/monthly. An all-zero policy keeps everything — prune is a no-op until configured. - Add app.Prune: scopes to a profile, plans over chains, and on --apply deletes leaf-inward (incrementals before base) so an interrupted prune leaves at worst a complete shorter chain. Per-dump failures are collected, not fatal; the run reports them and exits non-zero. - Rewire siphon prune to the engine: --profile, --keep-last, --max-age, --gfs-daily/weekly/monthly, dry-run by default, --apply to delete. - Add a retention config block (defaults + per-profile override that replaces wholesale) with fail-fast validation; precedence is CLI flags > profile > defaults > keep-everything. - Document in docs/RETENTION.md; update README and CHANGELOG. Unit-test the engine (GFS bucketing, union, chain grouping, keep-everything), the executor (leaf-inward order, scoping, collected failures), and the config (validation, override precedence).
|
Caution Review failedAn error occurred during the review process. Please try again later. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (8)
✅ Files skipped from review due to trivial changes (2)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughWalkthroughThis PR adds chain-aware retention planning, config-driven prune policy resolution, and CLI/app execution for dry-run or apply pruning. It also updates the README, changelog, and retention docs to describe the new policy shape and prune behavior. ChangesRetention pruning flow
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai full review |
✅ Action performedFull review finished. |
✅ Action performedFull review finished. |
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (2)
internal/config/retention_test.go (1)
27-53: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd coverage for an explicit empty profile override.
The key contract here is “profile block replaces defaults wholesale.” This test covers omitted vs non-empty overrides, but not the
retention: {}case that disables inherited defaults for a single profile.🧪 Suggested test shape
func TestEffectiveRetention_Precedence(t *testing.T) { def := &RetentionConfig{KeepLast: 7} prof := &RetentionConfig{KeepLast: 30} + emptyOverride := &RetentionConfig{} cfg := &Config{ Defaults: Defaults{Retention: def}, Profiles: map[string]ProfileConfig{ "prod": {Name: "prod", Retention: prof}, "staging": {Name: "staging"}, // no override + "dev": {Name: "dev", Retention: emptyOverride}, }, } @@ if got := cfg.EffectiveRetention("unknown"); got == nil || got.KeepLast != 7 { t.Errorf("unknown-profile effective = %+v, want defaults", got) } + if got := cfg.EffectiveRetention("dev"); got != emptyOverride { + t.Errorf("dev effective = %+v, want explicit empty override", got) + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/config/retention_test.go` around lines 27 - 53, Add a test case in TestEffectiveRetention_Precedence to cover an explicit empty retention override on a profile, using Config.EffectiveRetention and the existing prod/staging setup as reference. Create a profile with Retention set to an empty RetentionConfig and assert it returns a non-nil empty result instead of inheriting Defaults.Retention, while omitted retention still falls back to defaults. Keep the assertions aligned with the current contract that a profile block replaces defaults wholesale.internal/app/prune_test.go (1)
178-194: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a multi-member failure regression here.
This only covers a single-dump chain, so it will not catch the case where a failed leaf delete is followed by parent deletion. Seed
base -> inc1 -> inc2, failinc2.meta.json, and assert the older members are left intact.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/app/prune_test.go` around lines 178 - 194, Add a regression in TestPrune_CollectsDeletionFailures for a multi-member chain: instead of only the single dump case, seed a base -> inc1 -> inc2 sequence, configure the store to fail deleting inc2.meta.json, and assert Prune collects the failure without deleting the older parent members. Use the existing helpers like newRecordingStore, seedDump, Prune, and pruneDeps to locate and extend the test.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/app/prune.go`:
- Around line 84-90: The deletion loop in applyChainDeletion continues after a
failed d.Dumps.Delete call, which can leave an incremental orphaned after a
.meta.json failure. Change the loop so that once the first member deletion
fails, it records the error on ChainOutcome/PruneResult and stops deleting any
older members in that chain, preserving the complete-shorter-chain behavior.
In `@internal/cli/dumps.go`:
- Around line 162-175: The dumps renderer is marking every applied outcome as
fully “pruned” even when `oc.Errors` indicates a partial failure. Update the
logic in `RenderDumpPrune` so the verb only becomes `pruned` for outcomes that
succeeded without errors, and use a different message for partially failed
chains while still listing `oc.Errors`; keep `Apply`, `res.Outcomes`, and
`oc.Pruned`/`oc.Errors` as the key checks.
- Around line 85-109: Validate the CLI retention override values before invoking
app.Prune in the dumps command flow, since the policy built in the CLI can
currently accept invalid negative values from flags. Add upfront checks around
the override handling near resolveRetentionPolicy and the
keepLastSet/maxAgeSet/gfs*Set assignments so destructive pruning fails fast with
a clear error instead of passing an undefined policy into app.Prune. Keep the
validation aligned with the existing config-backed retention rules and ensure
invalid overrides are rejected before buildDeps and app.Prune are called.
In `@internal/config/config.go`:
- Around line 61-64: Reject negative retention.max_age values in the config
validation path: in the parsing logic for r.MaxAge, after time.ParseDuration
succeeds, add a check that the resulting duration is not less than zero and
return a validation error if it is. Update the validation in the config reader
around r.MaxAge so it still accepts valid durations but explicitly rejects
signed negatives like -1h, keeping the existing error handling style and the
same retention.max_age validation flow.
In `@internal/dumps/retention.go`:
- Around line 81-90: The chain ordering in GroupChains is only sorted by
newest() after building from a map, so equal timestamps still follow randomized
map iteration order. Update the newest-first sort on Chain (and any matching
sort in Plan) to use a stable secondary key like Chain.Root when newest() ties,
or remove the duplicate Plan sort if GroupChains already guarantees determinism.
Use the existing Chain.newest() comparator and the Root field to make keep_last
and GFS selection deterministic across runs.
- Around line 65-70: GroupChains is currently ordering Chain.Members by Created
alone, which can break the base-first/apply-order contract. Update the
grouping/sorting logic in GroupChains to build each chain’s order from
ParentID/BaseID topology first so the root/base always comes before its
incrementals, and use Created only as a secondary tie-breaker for ties. Keep the
Chain.Members ordering consistent with the documented behavior and the
prune/apply path expectations.
In `@README.md`:
- Around line 184-194: The prune command name in this README example is
incorrect and should match the documented CLI path. Update the retention section
text to use the actual command name from the commands table, `siphon dumps
prune`, and make sure any related mention of dry-run or `--apply` refers to that
same command so users are pointed to the right entrypoint.
---
Nitpick comments:
In `@internal/app/prune_test.go`:
- Around line 178-194: Add a regression in TestPrune_CollectsDeletionFailures
for a multi-member chain: instead of only the single dump case, seed a base ->
inc1 -> inc2 sequence, configure the store to fail deleting inc2.meta.json, and
assert Prune collects the failure without deleting the older parent members. Use
the existing helpers like newRecordingStore, seedDump, Prune, and pruneDeps to
locate and extend the test.
In `@internal/config/retention_test.go`:
- Around line 27-53: Add a test case in TestEffectiveRetention_Precedence to
cover an explicit empty retention override on a profile, using
Config.EffectiveRetention and the existing prod/staging setup as reference.
Create a profile with Retention set to an empty RetentionConfig and assert it
returns a non-nil empty result instead of inheriting Defaults.Retention, while
omitted retention still falls back to defaults. Keep the assertions aligned with
the current contract that a profile block replaces defaults wholesale.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 84741d26-28e7-45f4-830f-9d61aa016d6e
📒 Files selected for processing (11)
CHANGELOG.mdREADME.mddocs/RETENTION.mdinternal/app/prune.gointernal/app/prune_test.gointernal/cli/dumps.gointernal/config/config.gointernal/config/retention_test.gointernal/dumps/prune.gointernal/dumps/retention.gointernal/dumps/retention_test.go
- Apply post-review fixes to the retention cycle. - Order chain members topologically (base first, each child after its parent via ParentID), with Created only as a tie-breaker. Ordering by Created alone could place a child ahead of its base on tied or skewed timestamps, and the leaf-inward delete then removes the base before a surviving incremental — orphaning it. Stragglers from broken/cyclic links are appended, never dropped. - Stop deleting a chain's members at the first failure (return, not continue): once a leaf delete fails, deleting its ancestors would orphan it, so abort the chain and leave a complete shorter prefix. Other chains still proceed. - Make chain ordering deterministic: break equal newest() timestamps by Root so keep_last / GFS selection is reproducible (map iteration is randomized; a planner must not be). Same comparator used in Plan. - Reject negative retention.max_age in config validation: ParseDuration accepts "-1h", which would push the cutoff into the future and make the whole catalog deletable. - Reject negative CLI flag overrides before planning, mirroring the config-load validation that flags otherwise bypass. - Label a partially deleted chain "partially pruned", not "pruned". - Fix the command name in README and docs/RETENTION.md: it is "siphon dumps prune", not "siphon prune". - Add tests: topological member order on tied/skewed timestamps, deterministic chain order on tied timestamps, and that a chain's deletion stops (base untouched) when the leaf delete fails.
Summary by CodeRabbit
retention:configuration with defaults plus per-profile overrides, including precedence and “keep everything” behavior for empty/zero policies.--applydeletion behavior.